Method and apparatus for controlling display operations

ABSTRACT

A graphics processing system includes a graphics processor  1  that renders output frames that are written to a frame buffer in a memory  2  for display on a display  7.  Comparison and control hardware  5  of the graphics processing system operates to compare successive output frames that are being generated for display, and then controls one or more aspects of the way in which the display of the output frames generated by the graphics processor  1  is carried out. In one preferred embodiment, the rate at which the display device  7  is updated (refreshed) from the frame buffer is controlled on the basis of the output frame comparisons. In another preferred embodiment, the format used when storing the output frames in the frame buffer is selected on the basis of the output frame comparisons.

The present invention relates to the controlling of display operationswhere an output is to be displayed on a display device such as an LCDdisplay, and in particular to such arrangements where a data processingsystem, such as a graphics processing system, is providing an output tobe displayed on a display device.

As is known in the art, the output of an, e.g., graphics processingsystem that is to be displayed is usually written to a so-called “framebuffer” in memory when it is ready for display. The frame buffer is thenread by a display controller and output to a display device (which may,e.g., be a screen or printer) for display.

The reading of the frame buffer and its provision to the display deviceis a relatively expensive operation. For example, LCD displays aretypically refreshed at a constant high rate typically between 60-70 Hz.Each such “refresh” involves reading the complete frame buffer frommemory and writing the contents to the display. This involves, interalia, a lot of power hungry memory and display accesses.

The bandwidth of reading from and writing to the frame buffer can alsobe a significant system bandwidth and power cost, particularly in thecase of high definition (HD) displays.

One known technique for trying to reduce the power consumption ofdisplay operations is to reduce the display refresh rate under thecontrol of the application that is generating the output to bedisplayed. For example, if the application knows that the output framehas not changed or will not change for a while, it can signal thedisplay controller to defer, or reduce the rate of, refreshing thedisplay. However, these arrangements rely on the application and/oroperating system itself being able to perform the necessary displaycontrol.

The Applicants believe therefore that there remains scope forimprovements to display operations in data processing systems.

According to a first aspect of the present invention, there is provideda method of operating a data processing system in which a stream ofoutput frames to be displayed is generated by the data processing systemand written to a frame buffer for display on a display device, themethod comprising:

the data processing system comparing output frames to be displayed, andcontrolling at least one aspect of the way in which the output framesare provided for display on the display device on the basis of thecomparison.

According to a second aspect of the present invention, there is provideda data processing system comprising:

a data processor for generating a stream of output frames to bedisplayed;

a frame buffer for storing an output frame to be displayed;

a write controller for writing an output frame generated by the dataprocessor to the frame buffer;

a display device for displaying an output frame;

a display controller for reading an output frame from the frame bufferand for providing it to the display device for display; and

processing circuitry for comparing output frames to be displayed and forcontrolling at least one aspect of the way in which the output framesare provided for display on the display device on the basis of thecomparison.

According to a third aspect of the present invention, there is provideda display control apparatus for use in a data processing system in whichsuccessive output frames to be displayed are generated by the dataprocessing system and written to a frame buffer for display on a displaydevice, the apparatus comprising:

processing circuitry arranged to compare output frames to be displayed,and to control at least one aspect of the way in which the output framesare provided for display on the display device on the basis of thecomparison.

The present invention relates to arrangements in which a data processingsystem, such as a graphics processing system, produces a stream ofoutput frames to be displayed on a display device, such as a screen.However, in the present invention, one or more aspects of the way inwhich the display of output frames on the display device is carried outis controlled on the basis of a comparison of output frames to bedisplayed. As will be discussed further below, the Applicants haverecognised that by comparing output frames, a measure, for example, ofthe extent to which different output frames differ from each other canbe derived, and, moreover, that this information can then be used toadvantageously control the operation for displaying the output frames inuse.

For example (and again as will be discussed further below), if it isfound that the output frames are not changing, the update or refreshrate of the display device can be reduced without significantlydegrading the image as seen by the user, thereby reducing the power,etc., that will be consumed by the display operation.

Similarly, and again as will be discussed further below, it would bepossible to alter the format of the stored output frame data (e.g. asbetween a lower quality format requiring fewer bits of data and a higherquality format requiring more bits of data) depending upon the rate atwhich the display is found to be changing. This could then be used,e.g., to control the format that the output frames are stored in in theframe buffer (and hence the bandwidth needed for the display operations)dynamically in use.

Thus, the present invention can be used to reduce significantly thepower consumed and memory bandwidth used for display operations, ineffect by facilitating the identification of opportunities to performthe display operations in a more efficient manner.

Moreover, in contrast to the prior art schemes discussed above, thepresent invention does not depend upon any application or operatingsystem operation, will work for a variety of sources (and not, e.g.,only for specific applications), can provide a more immediate anddynamic response and control, and can provide greater power andbandwidth savings than known prior art systems.

The aspect or aspects of the way in which the output frames are providedfor display that is or are controlled in response to the output framecomparisons in the present invention can relate to any suitable anddesired aspect of the process for displaying the output frames. Forexample, as discussed above, it could (and, indeed, preferably does)relate to the update or refresh rate of the display and/or to the formatthat the output frames are stored in the frame buffer.

In a particularly preferred embodiment, the way that output frames arewritten to, and/or are read from, the frame buffer is controlled on thebasis of the output frame comparisons. Most preferably the rate ofwriting frames to and/or of reading frames from the frame buffer, and/orthe format in which the frames are written to and/or read from the framebuffer, is controlled on the basis of the frame comparisons.

In one particularly preferred embodiment, the update or refresh rate ofthe display device (i.e. the rate at which frames are read from theframe buffer and provided to the display device) is controlled on thebasis of the comparisons of the output frames that are generated by thedata processing system.

Most preferably, if it is determined by the output frame comparisonsthat the output image is not changing or is not changing very rapidly(e.g., and preferably, at less than a threshold rate), the displaydevice update (refresh) rate is reduced, and vice-versa. In this case,when it is determined that the output frames being generated for displayare only changing at a low rate, the image that is displayed on thedisplay device is updated at a lower rate, thereby reducing powerconsumption and bandwidth for the display operation.

Most preferably, if it is determined by the output frame comparison thata new output frame should be considered to be the same as the currentoutput frame being displayed, then the output display is not updated,but if it is determined by the output frame comparison that a new outputframe should be considered to be different to the output frame currentlybeing displayed, then the output display is updated.

By controlling the rate that the display device is updated in thismanner, unnecessary updates to the display device can be avoided oreliminated, thereby saving power and bandwidth in the display device,the frame buffer memory, the interconnect and the display controller,etc.

For example, if one considers first order effects of frame bufferaccesses only, and ignores on-chip interconnect, display controller, andvideo output power consumption, reducing the display refresh rate for anLCD display from 60 fps (frames per second) to 20 fps using the presentinvention may save 200 mW and 316 MB/s for HD and 75 mW and 120 MB/s for1024×768 resolution displays.

It is believed that such arrangements may be new and advantageous intheir own right. Thus, according to a fourth aspect of the presentinvention, there is provided a method of operating a data processingsystem in which a stream of output frames to be displayed is generatedby the data processing system and written to a frame buffer for displayon a display device, the method comprising:

the data processing system comparing output frames to be displayed, andcontrolling the rate at which the display device is updated from theframe buffer on the basis of the comparison.

According to a fifth aspect of the present invention, there is provideda data processing system comprising:

a data processor for generating a stream output frames to be displayed;

a frame buffer for storing an output frame to be displayed;

a write controller for writing an output frame generated by the dataprocessor to the frame buffer;

a display device for displaying an output frame;

a display controller for reading an output frame from the frame bufferand for providing it to the display device for display; and

processing circuitry for comparing output frames to be displayed and forcontrolling the rate at which the display device is updated from theframe buffer on the basis of the comparison.

According to a sixth aspect of the present invention, there is provideda display control apparatus for use in a data processing system in whichsuccessive output frames to be displayed are generated by the dataprocessing system and written to a frame buffer for display on a displaydevice, the apparatus comprising:

processing circuitry arranged to compare output frames to be displayed,and to control the rate at which the display device is updated from theframe buffer on the basis of the comparison.

As will be appreciated by those skilled in the art, these aspects andembodiments of the invention may and preferably do include any one ormore or all of the preferred and optional features of the inventiondescribed herein, as appropriate. Thus, for example, preferablysuccessive generated output frames are compared and if they aredetermined to be the same, the display device is not updated, andvice-versa.

In a preferred arrangement of these aspects and embodiments of theinvention, the display operation is configured to always update thedisplay device at a minimum rate (e.g. to always update the displayafter a predetermined time period since the last update), irrespectiveof the result of the frame comparisons. This will help to avoid thedisplayed image fading away or degrading too much as a consequence ofthe operation of the present invention (and the minimum rate ispreferably selected accordingly so as to achieve this).

Thus, in a particularly preferred embodiment, the display device isupdated (the frame buffer is read out to the display device) only whenthe output frame has been determined to change, subject to an overallminimum update rate (that is chosen, e.g., to avoid flicker). This willkeep the number of display device updates to a minimum whilst stillensuring a satisfactory displayed image.

A down counter may, for example, be used to trigger the “minimum rate”display updates.

It would also be possible to set a maximum update rate, if desired. Thismay be useful where, for example, the data processing system can produceoutput frames at a rate that is much higher than is needed for thedisplay device. Again, a “maximum update rate” down counter could beused to prevent display updates at too high a rate.

In a preferred embodiment of these arrangements of the presentinvention, the display device's backlight is also controlled to reducethe perception of flickering at lower update rates (by adjusting thebacklight to keep the display brightness constant). For example where,as discussed below, only a portion of the displayed image is updated,the older portions of the image may have faded, whilst the recentlyupdated portion of the image will be brighter. The backlight can becontrolled to minimise the disparity between these different portions ofthe display. Where the backlight can be controlled independently fordifferent portions of the display, the backlight may be controlled withfiner granularity to achieve this, if desired.

Although in one preferred embodiment it is the entire output frame thatis updated (or not) on the display device on the basis of the outputframe comparisons, if the display device supports the possibility ofupdating only part (but not all) of the displayed frame, then thepresent embodiment can equally be applied in respect of particularregions or parts of the output display only. Thus, in a preferredembodiment, the frame comparisons assess whether a particular part orparts of the output frame have changed, and then those particular partsof the frame on the display device are updated (or not) accordingly.This would then facilitate only updating on the display device thoseparts or regions of the displayed frame that have changed, rather thanhaving to update the whole displayed frame.

In another particularly preferred embodiment, the way that the outputframe is stored in in the frame buffer (such as, and preferably, theframe buffer format that is used) is controlled (selected) on the basisof the output frame comparisons. This may be instead of or in additionto varying the display device update rate on the basis of the outputframe comparisons.

In a particularly preferred such embodiment, the output frame is storedin a relatively higher or a relatively lower quality format on the basisof the output frame comparisons. Most preferably a lower quality formatis used where the output frame is determined to be changing a lot and/orrelatively rapidly (as in this case a lower quality format may beacceptable), but a higher quality format is used when the output framesare not changing so much and/or are determined to be static (as thiswill provide a higher quality display whose quality may be visible on arelatively static or unchanging display, but not really appreciablewhere the display is changing more rapidly).

In these arrangements, the lower quality format may, e.g., be a lossy ormore lossy format (e.g. have a higher compression ratio), whereas thehigher quality format may be, e.g., a less lossy format (have a lowercompression ratio) or lossless.

In one particularly preferred arrangement of these embodiments of theinvention, the lower quality format that is used is a YUV format orsimilar (which, as is known in the art, is a format that is often usedfor video where the image is constantly moving), such as YUV 4:2:0, andthe higher quality format is an RGB format or similar, such as RGB 8:8:8or RGB 10:10:10 (which is a much higher quality format than YUV 4:2:0for example, but whose additional quality tends only really to bevisible on static images such as when viewing a static computing desktopor user interface (UI)).

Other possible arrangements would be to use higher and lower framebuffer resolutions, higher and lower dynamic ranges (colour depth) forthe frame buffer, and other more or less lossy compression schemes. Inthe case of rendering the image at a lower resolution (e.g. when theimage is changing rapidly) the display controller or another systemcomponent may then scale the image to full size for display.

Another arrangement would be to perform partial frame updates of theframe in the frame buffer when a lower quality frame buffer is desired(acceptable). For example, the data processor could be configured togenerate only a part of the new frame each time, such that, for example,for a first new frame only the top left portion of the frame isgenerated, followed a frame later by the bottom right portion of theframe and so on until the whole frame has been generated anew. Othersuitable such schemes could generate appropriately only every other line(or column) of the frame when a new frame is generated. This will reducethe workload for generating the output frame in the first place, therebyreducing power consumption and data processor bandwidth.

For displays that support partial frame updates, portions of the frameonly could be sent to the display, again reducing power consumption andbandwidth.

These arrangements of the present invention therefore effectivelyfacilitate using a lower quality frame buffer format, thereby reducingpower consumption and bandwidth, when there would be less or noperceived benefit to using a higher quality frame buffer format (e.g.because the output display is changing rapidly such that the user willbe unable to perceive a great deal of detail in the displayed image inany event), but reverting to a higher quality image format (such as RGB)when the additional quality of the image will actually be appreciated bythe viewer (e.g. because the image is static).

Thus, by selectively and dynamically switching between different framebuffer formats based on the comparisons of the output frames, a systemthat has, for example, improved quality as compared to a system thatalways uses a YUV format frame buffer, but that reduces memory bandwidthand power consumption as compared to a system that always uses an RGBformat frame buffer, can be provided.

It is again believed that these arrangements may be new and advantageousin their own right. Thus, according to a seventh aspect of the presentinvention, there is provided a method of operating a data processingsystem in which a stream of output frames to be displayed is generatedby the data processing system and written to a frame buffer for displayon a display device, the method comprising:

the data processing system comparing output frames to be displayed, andselecting the way in which the output frames are stored in the framebuffer on the basis of the comparison.

According to an eighth aspect of the present invention, there isprovided a data processing system comprising:

a data processor for generating a stream of output frames to bedisplayed;

a frame buffer for storing an output frame to be displayed;

a write controller for writing an output frame generated by the dataprocessor to the frame buffer;

a display device for displaying an output frame;

a display controller for reading an output frame from the frame bufferand for providing it to the display device for display; and

processing circuitry for comparing output frames to be displayed and forselecting the way in which the output frames are stored in the framebuffer on the basis of the comparison.

According to a ninth aspect of the present invention, there is provideda display control apparatus for use in a data processing system in whichsuccessive output frames to be displayed are generated by the dataprocessing system and written to a frame buffer for display on a displaydevice, the apparatus comprising:

processing circuitry arranged to compare output frames to be displayed,and to select the way in which the output frames are stored in the framebuffer on the basis of the comparison.

As will be appreciated by those skilled in the art, these aspects andembodiments of the invention may and preferably do include any one ormore or all of the preferred and optional features of the inventiondescribed herein, as appropriate. Thus, for example, the format to beused when storing the output frames in the frame buffer is preferablyselected on the basis of the frame comparisons, and the systempreferably selects between a lower quality frame buffer format, such asYUV, and a higher quality format such as RGB.

In a particularly preferred arrangement of these aspects and embodimentsof the invention, when the output frame comparisons indicate that theoutput frame is changing at greater than a particular, preferablypredetermined rate, the output frames are preferably stored in the framebuffer in a lower quality fashion, e.g., in a lower quality format, butwhen the rate of changing of the output frames drops (is lower) (e.g.and preferably changes at less than a particular, preferablypredetermined rate (which may be same or a different rate to thresholdrate for using a lower quality frame buffer format)) and/or the imagebecomes static, the output frames are stored in the frame buffer in ahigher quality fashion, e.g., format (and vice-versa).

In these aspects and embodiments of the present invention, the dataprocessing system and/or data processor could be controlled to generateits output frames in the desired frame buffer format (or other fashion)in the first place (for storage then in the frame buffer in that desired(appropriate) format or fashion). Alternatively, there could be someform of format conversion or processing stage or element (circuitry)that converts the output frames, if necessary, to the desired framebuffer format, etc., before they are stored in the frame buffer. (Forexample, the output frame could be generated in one format by the dataprocessor but then converted to the desired frame buffer format beforeit is stored in the frame buffer).

In these arrangements, the stored output frame preferably has associatedwith it (e.g. stored with it or in association with it) controlinformation to allow the stored frame in the frame buffer to be providedin the appropriate format to the display device (where that isnecessary). This control information may, e.g., indicate what format theframe has been stored in in the frame buffer, and/or may indicate, orcomprise, an algorithm for converting the data in the frame buffer tothe desired final display format.

Then, when the frame is read from the frame buffer for display, thiscontrol information is preferably also read (e.g. by the displaycontroller) to allow the frame buffer format and/or necessary conversionprocess to be determined so that the stored frame can be convertedappropriately (and if necessary) to the desired format for the displaydevice.

Again, although in one preferred arrangement of these embodiments of theinvention it is the entire output frame that is stored in the desiredframe buffer format, etc., it would equally be possible, as for thedisplay updating, to apply these techniques in respect of particularportions or regions of the (stored) output frame only. Thus, forexample, different frame buffer formats could be used for differentportions of a frame (and in one preferred embodiment this is done),e.g., depending upon the determined rate of change of the respectiveportions of the frame.

Thus, in a preferred embodiment, the frame comparisons assess whether aparticular part or parts of the output frame have changed, and then theway in which those particular parts of the frame are stored in the framebuffer (such as the frame buffer format to be used) is selectedaccordingly. This would then facilitate varying the frame buffer format,for example, for those parts or regions of the displayed frame that havechanged, without having to change the format for the whole frame buffer.

Other aspects of the writing and reading of the output frames to andfrom the frame buffer could also or instead be controlled on the basisof the comparison of the output frames generated by the data processingsystem, if and as desired.

For example, the level of anti-aliasing (e.g. of multisampling orsupersampling) being used could be varied based on the output framecomparisons, e.g. to use better anti-aliasing (thereby giving higher(perceived) image quality) when it is determined that the image is onlychanging slowly and/or is static and vice-versa.

It would also or instead be possible to vary the rate at which theoutput frames are produced by the data processing system based on theoutput frame comparisons, for example, to reduce the rate of outputframe generation (thereby reducing power consumption) when it isdetermined that the image is only changing slowly and/or is static andvice-versa.

The output frames that are generated by the data processing system canbe compared in any suitable and desired manner. As discussed above, thecomparison is preferably so as to determine or at least estimate whetherthe frames are changing or have changed, and/or the rate at which theframes are changing. Any comparison process that is able to assess thismay be used.

Thus, in a particularly preferred embodiment, the comparison process isso as to assess or estimate (and is to be used to assess or estimate)the correlation between successive, and/or sequences of, output framesgenerated by the data processing system (i.e. the extent to which theoutput frames are similar to each other). Thus, the comparison ispreferably so as to determine whether one frame is the same as (or atleast sufficiently similar to) the other (or another) frame or not.

Then, if the comparison process indicates that successive frames are thesame (the correlation is high) that would suggest, e.g., that the imageis static for a period of time (in which case, as discussed above, itmay be possible to reduce the display update rate, for example), andvice-versa.

Preferably the comparison process uses some form of threshold or tidevalue or values (parameter or parameters) to determine whether theoutput frames should be considered to be changing or not (to bedifferent or to be static) for the purposes of the control operation ofthe present invention. Then, if the comparison indicates a value to oneor other side of the appropriate threshold, the display operation willbe controlled accordingly.

In a particularly preferred embodiment, the comparison process comparesone output frame with its immediately preceding output frame (i.e. suchthat adjacent frames in the stream of output frames generated by thedata processing system are compared with each other). This is preferablydone by comparing a newly generated output frame with the (appropriate)frame that is currently stored in the frame buffer.

Thus, in a preferred embodiment a newly generated output frame iscompared with an output frame already stored in the frame buffer, mostpreferably so as to determine whether the new output frame should beconsidered to be the same as (or at least sufficiently similar to) thealready stored output frame or not.

The comparison is preferably repeated for each new output frame that isgenerated, i.e. such that there will be successive comparisons of pairsof output frames as the output frames are generated by the dataprocessing system.

In this case, the system could, e.g., count how many compared frames areconsidered to be the same as each other (or to differ from each other),e.g. in a given period and then compare that count to a threshold valueto determine whether the image should be considered to be static(unchanging) or not.

Most preferably, the comparison process determines whether a respectivepair of frames being compared should be considered to be the same aseach other or not, and the display control is then performed on thebasis of that assessment, e.g., to update display device if the framesare considered to be different but not otherwise, and so on. This willthen provide a dynamic and “instant” response to the output frames thatare being produced.

Other arrangements would, of course, be possible. For example, a seriesof more than two output frames could be compared with each other ifdesired.

The comparison process preferably compares some or all of the content ofeach output frame that is being compared.

In a particularly preferred embodiment, the comparison is performed bycomparing information representative of and/or derived from the contentof one output frame with information representative of and/or derivedfrom the other output frame, e.g., and preferably, to assess thesimilarity or otherwise of the output frames.

The Applicants have recognised that in many data processing systemarrangements, an output frame for display will be generated and storedin the frame buffer on a data-block-by-data-block basis, rather thandirectly as a single, overall, output “frame”.

This will be the case, for example, and as will be appreciated by thoseskilled in the art, in a tile-based graphics processing system, in whichcase each block of data that is generated and stored in the frame buffermay correspond to a “tile” that the rendering process of the graphicsprocessor produces.

(As is known in the art, in tile-based rendering, the two dimensionaloutput array or frame of the rendering process (the “render target”)(e.g., and typically, that will be displayed to display the scene beingrendered) is sub-divided or partitioned into a plurality of smallerregions, usually referred to as “tiles”, for the rendering process. Thetiles (sub-regions) are each rendered separately (typically one afteranother). The rendered tiles (sub-regions) are then recombined toprovide the complete output array (frame) (render target), e.g. fordisplay.

Other terms that are commonly used for “tiling” and “tile based”rendering include “chunking” (the sub-regions are referred to as“chunks”) and “bucket” rendering. The terms “tile” and “tiling” will beused herein for convenience, but it should be understood that theseterms are intended to encompass all alternative and equivalent terms andtechniques.)

Where the present invention is implemented in a data processing systemin which the overall, “final” output frame of the data processing systemis stored in the frame buffer on a block-by-block basis, then in aparticularly preferred embodiment the output frames are compared bycomparing respective data blocks of the output frames. In other words,the comparison process preferably comprises comparing blocks of datarepresenting particular regions of the respective output frames witheach other.

In a particularly preferred embodiment of these arrangements of thepresent invention, the data blocks are compared by comparing a datablock that is newly generated by the data processing system (for a newoutput frame) with at least one data block of a previous output frame,e.g., and preferably, that is already stored in the frame buffer.

Thus, in a particularly preferred embodiment of the present invention,the data processing system comprises a data processing system in whichthe output frame is stored in the frame buffer by writing blocks of datarepresenting particular regions of the output frame to the frame buffer,and the comparison process comprises comparing a block of data that isto be written to the frame buffer to at least one block of data alreadystored in the frame buffer. This will then preferably be repeated forplural data blocks making up the respective output frame beinggenerated.

The blocks of data that are considered and compared in these embodimentsof the present invention can each represent any suitable and desiredregion (area) of the overall output frame that is to be stored in theframe buffer. So long as the overall output frame is divided orpartitioned into a plurality of identifiable smaller regions eachrepresenting a part of the overall output frame, and that canaccordingly be represented as blocks of data that can be identified andcompared, then the sub-division of the output frame into blocks of datacan be done as desired.

Each block of data preferably represents a different part (sub-region)of the overall output frame (although the blocks could overlap ifdesired). Each block should represent an appropriate portion (area) ofthe output frame, such as a plurality of data positions within theframe. Suitable data block sizes would be, e.g., 8×8, 16×16 or 32×32data positions in the output frame.

In one particularly preferred embodiment, the output frame is dividedinto regularly sized and shaped regions (blocks of data), preferably inthe form of squares or rectangles. However, this is not essential andother arrangements could be used if desired.

Where the data processor is a tile-based graphics processor, then in aparticularly preferred embodiment, each data block that is comparedcorresponds to a rendered tile that the graphics processor produces asits rendering output. This is a particularly straightforward way ofimplementing these arrangements of the present invention, as thegraphics processor will generate the rendering tiles directly, and sothere will be no need for any further processing to “produce” the datablocks that will be considered and compared. In this case therefore, aseach rendered tile generated by the graphics processor is to be writtento the frame buffer, it will preferably be compared with a rendered tileor tiles already stored in the frame buffer.

In these arrangements of the present invention, the (rendering) tilesthat the output frame is divided into for rendering purposes can be anydesired and suitable size or shape. The rendered tiles are preferablyall the same size and shape, as is known in the art, although this isnot essential. In a preferred embodiment, each rendered tile isrectangular, and preferably 16×16, 32×32 or 8×8 sampling positions insize.

In a particularly preferred embodiment of these arrangements of thepresent invention, the comparison may be, and preferably is, also orinstead performed using data blocks of a different size and/or shape tothe tiles that the rendering process operates on (produces).

For example, in a preferred embodiment, a or each data block that isconsidered and compared may be made up of a set of plural “rendered”tiles, and/or may comprise only a sub-portion of a rendered tile. Inthese cases there may be an intermediate stage that, in effect,“generates” the desired data block from the rendered tile or tiles thatthe graphics processor generates.

In one preferred embodiment, the same block (region) configuration (sizeand shape) is used across the entire output frame. However, in anotherpreferred embodiment, different block configurations (e.g. in terms oftheir size and/or shape) are used for different regions of a givenoutput frame. Thus, in one preferred embodiment, different data blocksizes may be used for different regions of the same output frame.

Where the comparison process involves comparing respective blocks ofdata, then the comparison is, similarly, preferably so as to determinewhether the new data block is the same as (or at least sufficientlysimilar to) the already stored data block or not, and preferablycomprises comparing some or all of the content of the new data blockwith some or all of the content of the already stored data block.

In a particularly preferred embodiment, the comparison is performed bycomparing information representative of and/or derived from the contentof the new output data block with information representative of and/orderived from the content of the stored data block. In such a case, theinformation representative of the content of each data block ispreferably in the form of a “signature” for the data block which isgenerated from or based on the content of the data block. (Such a datablock content “signature” may comprise, e.g., and preferably, anysuitable set of derived information that can be considered to berepresentative of the content of the data block, such as a checksum, aCRC, or a hash value, etc., derived from (generated for) the data block.Suitable signatures would include standard CRCs, such as CRC32, or otherforms of signature such as MD5, SHA-1, etc . . . )

Thus, in a particularly preferred arrangement of these embodiments ofthe present invention, a signature, such as a CRC value, is generatedfor each data block that is to be written to the frame buffer (e.g. andpreferably, for each output rendered tile that is generated) and thecomparison process comprises comparing the signatures of the respectivedata blocks.

The signatures for the data blocks (e.g. rendered tiles) that are storedin the frame buffer should be stored appropriately. Preferably they arestored with the frame buffer. Then, when the signatures need to becompared, the stored signature for a data block can be retrievedappropriately.

The current, completed (newly generated) data block (e.g. rendered tile)(e.g., and preferably, its signature) can be compared with one, or withmore than one, data block that is already stored in the frame buffer. Itshould be compared with the equivalent data block (or blocks, ifappropriate) already stored in the frame buffer (i.e. the data block(s)in the frame buffer occupying the same position (the same data block(e.g. tile) position)) as the completed, new data block is to be writtento).

In one preferred embodiment, all the data blocks making up the outputframe being considered are compared. However, in other arrangements somebut not all of the blocks are compared. This could be the case where,for example, only a portion of the output frame is to be considered, orwhere, for example, it is known that a certain portion of the frame isalways going to change or be static (such that that portion doesn't needchecking). In the latter case, information about whether portions of theimage are static or dynamic could, e.g., be provided to the comparisonprocessing circuitry, for example by the application in question.

The former case could occur, for example, where, as discussed above, thedisplay arrangement supports updating only part but not all of thedisplayed frame. In this case the comparisons could be carried out inrespect of particular regions of the output frame only, i.e. for thoseparts of the frame that are being, or that are potentially to be,updated.

In a particularly preferred embodiment, the number of data blocks thatare compared with a stored data block or blocks for respective outputframes is varied, e.g., and preferably, on a frame-by-frame, or oversequences of frames, basis. This may be particularly applicable where,for example, only portions of the display are to be updated or differentregions are to be stored in different formats. It is preferably based onthe expected correlation (or not) between successive output frames.

In a preferred embodiment, it is possible for the software applicationthat is controlling the display to indicate and control which regions ofthe frame are compared, e.g., in particular, and preferably, to indicatewhich regions of the frame the data comparison process should beperformed for. This would then allow the comparison calculation to be“turned off” by the application for regions of the frame the application“knows” will be static.

This may be achieved as desired. In a preferred embodiment registers areprovided that enable/disable data block (e.g. rendered tile) signaturecomparison for frame regions, and the software application then sets theregisters accordingly (e.g. via the graphics processor driver). Thenumber of such registers may be chosen, e.g., as a trade-off between theextra logic required for the registers, the desired granularity ofcontrol, and the potential savings from being able to disable thecomparison calculations.

It would also be possible to use information in registers elsewhere inthe system, for example in a graphics processor's scissor or stencilregisters, to determine where comparisons need to be made, if desired.

The data block comparisons are preferably used, as discussed above, toestimate or determine the similarity or not of (the correlation between)the output frames being compared. This is preferably estimated on thebasis of the number of data blocks that are found to match (that areconsidered to match) and/or that are found to mis-match between the twoframes. Most preferably the number of mismatching (unmatching) datablocks in the frames being compared is counted and if the number ofmis-matching blocks exceeds a particular, preferably predetermined,threshold value, the two frames are considered not to match (i.e. theimage is considered to have changed), and vice-versa (as between thosetwo output frames).

Thus, in a particularly preferred embodiment, the number of data blocksfound to match or to mismatch in the two output frames being compared isused as a measure of the correlation of the frames and to determine howthe relevant aspect of the display operation should be controlled. Forexample, and preferably, if the count of mismatching blocks exceeds athreshold value, the display is updated, but not otherwise. Similarly,if the count of mismatching blocks is above a threshold value (whichwould preferably be a higher value than the threshold to trigger adisplay update), then preferably a lower quality (less accurate) framebuffer format is used.

The count of mismatching (or matching) data blocks may be performed asdesired, for example by using a standard counter, using a bitmap thatcan be analysed, or using a hierarchical count (e.g. a count per tile,then a level up of a count for a group of four tiles, and so on, all theway up to a single count per frame). A hierarchical count would allowportions of the frame that have matches or mismatches to be searchedrapidly.

In one preferred embodiment, where the number of matching blocks isbelow a threshold value, the mismatching blocks are analysed todetermine the magnitude of the visual differences, and the displayoperation is controlled on the basis of the determined magnitude of thevisual differences.

Where the present invention is to be used with a double-buffered framebuffer, i.e. a frame buffer which stores two output frames concurrently,e.g. one being displayed and one that has been displayed and istherefore being written to as the next output frame to display, then thecomparison process of the present invention preferably compares thenewly generated output frame with the output frame in the frame bufferthat is currently being displayed.

The comparison process and consequent display control may be implementedin an integral part of the data processor, or there may, e.g., be aseparate hardware element or elements that perform these functions.

In a particularly preferred embodiment, there is a “display control”hardware element or elements that carry out the comparison process andcontrol the relevant display operation.

In one preferred embodiment, this hardware element or elements isseparate to the data processor, and in another preferred embodiment isintegrated in (part of) the data processor. Thus, in one preferredembodiment, the comparison element etc., is part of the data processoritself, but in another preferred embodiment, the data processing systemcomprises a data processor, and a separate “display control” unit orunits or element or elements that perform the frame comparison, etc.

The present invention can be used wherever a data processing system isproviding output frames intended to form an image for display.

Similarly, although in one particularly preferred embodiment the presentinvention is used in conjunction with a graphics processing system (andthe data processor is a graphics processor), the principles of thepresent invention can equally be applied to other systems that willproduce output frames for display in a similar manner to graphicsprocessing systems. Thus the present invention may equally be used, forexample, for video processing (and video processing analogously operateson blocks of data analogous to the tiles in tile-based graphicsprocessing (in which case the data processor will be a videoprocessor)), and for composite image processing (again, the compositionframe buffer can be processed as distinct blocks of data).

In such arrangements, in a similar manner to that discussed above inrelation to tile-based graphics processing systems, the data blocks thatare compared (where that is done) may comprise, e.g., video data blocksproduced by the video processing system (a video processor), and/orcomposite frame tiles produced by a composition processing system, etc.

The present invention also extends to the provision of a particularhardware element for performing the comparison and consequent displaycontrol determination of the present invention. As discussed above, thishardware element (logic) may, for example, be provided as an integralpart of a, e.g., graphics or other processor, or may be a standaloneelement. It may be a programmable or dedicated hardware element.

Similarly, the frame buffer that the output frames are to be written tomay comprise any suitable such buffer and may be configured in anysuitable and desired manner in memory. For example, it may be an on-chipbuffer or it may be an external buffer (and, indeed, may be more likelyto be an external buffer (memory), as will be discussed below).Similarly, it may be dedicated memory for this purpose or it may be partof a memory that is used for other data as well.

The display may similarly be any suitable form of display. In onepreferred embodiment it is an LCD display.

The various functions of the present invention can be carried out in anydesired and suitable manner. For example, the functions of the presentinvention can be implemented in hardware or software, as desired. Thus,for example, the various units and elements, etc., of the invention maycomprise a suitable processor or processors, functional units,circuitry, processing circuitry, logic, processing logic, microprocessorarrangements, etc., that are operable to perform the various functions,etc., such as appropriately dedicated hardware elements and/orprogrammable hardware elements that can be programmed to operate in thedesired manner.

In a preferred embodiment the data graphics, e.g., processor and/ordisplay control unit or units is implemented as a hardware element orelements (e.g. ASIC). Thus, in another aspect the present inventioncomprises a hardware element including the apparatus of, or operated inaccordance with the method of, any one or more of the aspects of theinvention described herein.

It should also be noted here that, as will be appreciated by thoseskilled in the art, the various functions, etc., of the presentinvention may be duplicated and/or carried out in parallel on a givenprocessor.

The present invention is applicable to any suitable form orconfiguration of graphics processor and renderer, such as processorshaving a “pipelined” rendering arrangement (in which case the rendererwill be in the form of a rendering pipeline). It is particularlyapplicable to tile-based graphics processors and graphics processingsystems.

As will be appreciated from the above, the present invention isparticularly, although not exclusively, applicable to 2D and 3D graphicsprocessors and processing devices, and accordingly extends to a 2Dand/or 3D graphics processor and a 2D and/or 3D graphics processingplatform including the apparatus of, or operated in accordance with themethod of, any one or more of the aspects of the invention describedherein. Subject to any hardware necessary to carry out the specificfunctions discussed above, such a 2D and/or 3D graphics processor canotherwise include any one or more or all of the usual functional units,etc., that 2D and/or 3D graphics processors include.

It will also be appreciated by those skilled in the art that all of thedescribed aspects and embodiments of the present invention can include,as appropriate, any one or more or all of the preferred and optionalfeatures described herein.

The methods in accordance with the present invention may be implementedat least partially using software e.g. computer programs. It will thusbe seen that when viewed from further aspects the present inventionprovides computer software specifically adapted to carry out the methodsherein described when installed on data processing means, a computerprogram element comprising computer software code portions forperforming the methods herein described when the program element is runon data processing means, and a computer program comprising code meansadapted to perform all the steps of a method or of the methods hereindescribed when the program is run on a data processing system. The dataprocessing system may be a microprocessor, a programmable FPGA (FieldProgrammable Gate Array), etc.

The invention also extends to a computer software carrier comprisingsuch software which when used to operate a graphics processor, rendereror microprocessor system comprising data processing means causes inconjunction with said data processing means said processor, renderer orsystem to carry out the steps of the methods of the present invention.Such a computer software carrier could be a physical storage medium suchas a ROM chip, CD ROM or disk, or could be a signal such as anelectronic signal over wires, an optical signal or a radio signal suchas to a satellite or the like.

It will further be appreciated that not all steps of the methods of theinvention need be carried out by computer software and thus from afurther broad aspect the present invention provides computer softwareand such software installed on a computer software carrier for carryingout at least one of the steps of the methods set out herein.

The present invention may accordingly suitably be embodied as a computerprogram product for use with a computer system. Such an implementationmay comprise a series of computer readable instructions either fixed ona tangible medium, such as a computer readable medium, for example,diskette, CD ROM, ROM, or hard disk, or transmittable to a computersystem, via a modem or other interface device, over either a tangiblemedium, including but not limited to optical or analogue communicationslines, or intangibly using wireless techniques, including but notlimited to microwave, infrared or other transmission techniques. Theseries of computer readable instructions embodies all or part of thefunctionality previously described herein.

Those skilled in the art will appreciate that such computer readableinstructions can be written in a number of programming languages for usewith many computer architectures or operating systems. Further, suchinstructions may be stored using any memory technology, present orfuture, including but not limited to, semiconductor, magnetic, oroptical, or transmitted using any communications technology, present orfuture, including but not limited to optical, infrared, or microwave. Itis contemplated that such a computer program product may be distributedas a removable medium with accompanying printed or electronicdocumentation, for example, shrink wrapped software, pre loaded with acomputer system, for example, on a system ROM or fixed disk, ordistributed from a server or electronic bulletin board over a network,for example, the Internet or World Wide Web.

A number of preferred embodiments of the present invention will now bedescribed by way of example only and with reference to the accompanyingdrawings, in which:

FIG. 1 shows schematically a graphics processing system that can beoperated in accordance with the present invention;

FIG. 2 shows schematically and in more detail a first embodiment of thecomparison and control hardware unit and display controller shown inFIG. 1;

FIG. 3 shows schematically how relevant data may be stored in memory inan embodiment of the present invention;

FIGS. 4 and 5 show schematically and in more detail a second embodimentof the comparison and control hardware unit and display controller shownin FIG. 1.

Like reference numerals are used for like components throughout thefigures, unless otherwise indicated.

A number of preferred embodiments of the present invention will now bedescribed. These embodiments will be described primarily with referenceto the use of the present invention in a graphics processing system,although, as noted above, the present invention is applicable to otherdata processing systems which produce output frames for display, such asvideo processing systems, composite image processing systems, etc.

Similarly, the following embodiments will be described primarily withreference to the comparison of output frames by comparing rendered tilesgenerated by a tile-based graphics processor, although again and asnoted above, the present invention is not limited to such arrangements.

FIG. 1 shows schematically an arrangement of a graphics processingsystem that is in accordance with the present invention.

The graphics processing system includes, as shown in FIG. 1, atile-based graphics processor or graphics processing unit (GPU) 1,which, as is known in the art, will, inter alia, produce tiles of anoutput frame intended to be displayed on a display device, such as ascreen or printer.

As is known in the art, in such an arrangement, once a tile has beengenerated by the graphics processor 1, it would then normally be writtento a frame buffer in a memory 2 (which memory may be DDR-SDRAM) via aninterconnect 3 which is connected to a memory controller 4. Sometimelater the frame buffer will, e.g., be read by a display controller 6 andoutput to a display 7. In the present embodiment, the display device 7is an LCD screen (other arrangements would, of course, be possible).FIG. 1 also shows for completeness a host CPU 8.

In the present embodiment, and in accordance with the present invention,the above process is modified by the use of a comparison and controlunit 5, which controls one or more aspects of the way in which thedisplay of the output frames formed by the tiles generated by thegraphics processor 1 is carried out.

In essence, and as will be discussed further below, the comparison andcontrol hardware 5 operates to compare successive output frames that arebeing generated for display, and then controls, in the presentembodiment, the rate at which the display device 7 is updated(refreshed) from the frame buffer on the basis of those comparisons. Ineffect, the comparison and control hardware assesses whether successiveoutput frames differ from each other, and if they do not, the display 7is not updated (subject to a predetermined minimum update rate).

In this way, the present embodiment can avoid relatively costly updatesto the display device 7 when the output frame is not actually changingfrom one frame to the next. This can save a significant amount ofbandwidth and power consumption in relation to the display updateoperation.

FIG. 2 shows the comparison and control hardware unit 5 and the displaycontroller 6 that are used in the present embodiment in more detail. Forclarity purposes, FIG. 2 does not show the interconnect 3 or the memorycontroller 4, but it will be appreciated that the necessarycommunication between the comparison and control hardware unit 5, memory2 and display controller 6 shown in FIG. 2 is performed appropriatelyvia the interconnect 3 and memory controller 4.

In this embodiment, the comparison and control hardware 5 operates togenerate for each tile a signature representative of the content of thetile and then compares that signature with the signature of thecorresponding tile already stored in the frame buffer to see if thesignatures match. Thus, the output frames are compared by comparingrendered tiles generated by the graphics processor 1.

FIG. 3 shows an exemplary memory layout for storing the frame buffer andcorresponding tile signatures in the memory 2. The tiles making up theframe are stored in one portion 10 of the memory 2 (thus forming the“frame buffer”) and the associated signatures for the tiles making upthe frame are stored in another portion 11 of the memory. (Otherarrangements would, of course, be possible.)

As shown in FIG. 2, tile data 25 received by the comparison and controlhardware unit 5 from the graphics processor 1 is passed both to a buffer21 which temporarily stores the tile data while the signature generationand comparison process takes place, and a signature generator 20.

The signature generator 20 operates to generate the necessary signaturefor the tile. In the present embodiment the signature is in the form ofa 64-bit CRC for the tile. (Other signature generation functions andother forms of signature such as hash functions, etc., could also orinstead be used, if desired. It would also, for example, be possible togenerate a single signature for an RGBA tile, or a separate signaturefor each colour plane. Similarly, colour conversion could be performedand a separate signature generated for each of Y, U and V.)

Once the signature for the new tile has been generated, it is passed toa signature comparator 23, which operates to compare the signature ofthe new tile with the signature of the tile already in the frame bufferat the tile position for the tile in question.

The signatures for plural tiles from the previous frame are cached in asignature buffer 22 (this buffer may be implemented in a number of ways,e.g. buffer or cache) of the comparison and control hardware unit 5 tofacilitate their retrieval in operation of the system, and so thesignature comparator 23 fetches the relevant signature from thesignature buffer 22 if it is present there (or triggers a fetch of thesignature from the main memory 2, as is known in the art), and comparesthe signature of the previous frame's tile with the signature receivedfrom the signature generator to see if there is a match.

If the signatures are found not to match, then a counter 26 is updated.The counter 26 thus keeps a count of how many tiles have been found notto match for a respective pair of output frames being compared. Thecounter 26 is reset to zero each time a new output frame is started.

The count of mis-matching tiles for the current frame that is beingmaintained by the counter 26 is compared to a predetermined thresholdcount value or tidemark in a comparator 27. If the count of mis-matchingtiles for the output frames being compared is found to exceed thethreshold value, then it is assumed that the two output frames beingcompared are different, i.e. the image has been modified as between thetwo output frames being compared, and so an appropriate “frame-modified”indicator (flag) is set in a register 28. The “frame-modified” indicatoris provided as an input to a control unit 29 of the display controller6.

The control unit 29 operates to trigger an update of the display 7 fromthe frame buffer in the memory 2 if the “frame-modified” indicator is inits “set” (i.e. “do an update”) state when the new output frame has beencompleted in the frame buffer.

To facilitate this operation, the frame-modified indicator is resetwhenever the frame buffer is output to the display device 7, and then,as discussed above, changed to its set state when a sufficient number oftiles have been found to differ between the two output frames beingcompared.

Then, once a new output frame has been finished and is stored in theframe buffer, the controller 29 checks the state of the frame-modifiedindicator, and triggers the updating of the new output frame from theframe buffer to the display 7 to update the display if theframe-modified indicator is set to show that the new frame differs fromthe previous frame.

The number of the tiles that must differ to trigger a determination thatthe output frames differ (and thus that the display should be updated)can be set as desired, but should, for example, comprise a suitablysmall percentage of the tiles that make up an output frame.

In one preferred arrangement it is assumed that the output frames differ(i.e. a display update is triggered) if one pair of tiles being comparedis found to differ (i.e. such that, in effect, the threshold number oftiles that must differ to trigger a frame update is one). In this case,the counter 26 and threshold comparator 27 could be eliminated, and thecomparator 23 could be configured simply to set the frame-modifiedindicator to trigger a display update as soon as it is found that onepair of the tiles making up the output frames being compared differ fromeach other.

The control unit 29 of the display controller 6 triggers a displayupdate by providing a “fetch” control signal to a further control unit30 that controls the fetching of new frames from the frame buffer forprovision to the display device 7 in dependence upon the control inputsit receives.

In this way, the updating of output frames to the display device 7 iscontrolled in dependence upon the result of the comparison of the outputframes by the comparison and control hardware 5 and in particular on thebasis of whether the two output frames are to be considered to be thesame or not.

As shown in FIG. 2, the control unit 30 of the display controller 6controls a fetch element 31 which will read an output frame from theframe buffer in the memory 2 and provide that output frame via theappropriate display controller logic to the display device 7 fordisplay. In this way, the control unit 30 controls the updating (or not)of a new frame to the display 7.

The control unit 30 receives two inputs that can trigger it to trigger aframe update to the display 7. The first such input is the input fromthe control unit 29 based on the frame-modified indicator from theregister 28. The other input to the controller 30 that can trigger it tocause a display update is from a minimum update rate down counter 32.

The minimum update rate down counter 32 is set to a predetermined valuespecified in a minimum update rate register 33 each time a frame isoutput to the display device 7 for display, and thereafter counts downfrom that value until a new output frame is provided to the displaydevice 7 (at which point the minimum update rate down counter is resetto its initial value). If the minimum update rate down counter 32 everreaches zero, it signals the controller 30 to trigger the writing of anoutput frame to the display device 7. In this way, the minimum updaterate down counter 32 ensures that the display device 7 is updated at aminimum refresh rate, irrespective of the results of the output framecomparisons.

The display controller may also have a maximum refresh rate down counter(not shown), which is similarly reset to a predetermined value wheneveran output frame is written to the display device 7 and then counts downto zero thereafter, with the display controller being configured toprevent any further updates to the display device 7 until the maximumrefresh rate down counter has reached zero. This will stop the displaydevice 7 being updated at too high a rate in a situation where, forexample, the graphics processor 1 is able to generate output frames atmuch higher rates than in practice are needed for the display device 7.

The initial values that minimum update rate down counter 32 and themaximum update rate down counter (where present) are set to canpreferably be set and altered automatically, e.g. and preferablydepending upon whether the system is tethered or battery operated andupon the condition of the battery. This is preferably done by providingsignalling between the display controller 6 and a power management unitof the system.

The display controller signalling logic necessary to implement thepresent embodiment may be integrated in the display controller orexternal to the display controller, as desired.

Instead of, as shown in FIG. 2, using counters and logic, etc.,integrated into the comparison and control hardware 5 and displaycontroller 6, it would be possible to configure the comparison andcontrol hardware 5 to, for example, generate an interrupt, and then havethe display update rate control and process performed in software, withthe processor (e.g. the CPU 8) then signalling to the display controller6 when to update the display, if desired.

The backlight of the LCD display device 7 may be controlled to reducethe effect of flickering (which may occur when low update rates arebeing used), e.g. by adjusting the backlight to keep the displaybrightness constant as the screen fades away. For devices with anintegrated display, for example, the characteristics of the display(such as how the colour intensity changes if it isn't refreshed) can beused to determine how the image will fade and thus how the backlightshould be controlled to reduce flicker.

As can be seen from the above, in the present embodiment, the displaycontroller 6 does simply not update the LCD display device 7 at theusual constant rate of 60-70 Hz. Instead, differences are detectedbetween the current and subsequent output frames, and if the frames aredetermined to be the same, the display isn't refreshed, subject to stillrefreshing the display at a minimum rate to avoid the image fading awayor degrading.

In this way, unnecessary refreshes of the display device 7 are avoided,and when the image is static or only changing at a relatively slow rate,the image on the display is updated at a lower rate. This saves powerand bandwidth in the display device 7, the frame buffer memory, theinterconnect, and the display controller 6, etc.

Furthermore, this is achieved without any modification or particularoperation of the application or operating system, can provide animmediate response whenever the output frame changes, and can be used toensure as lower an update rate as possible without flickering or slowresponse.

As well as operating in the above manner to reduce the refresh (update)rate of the display device 7, the present embodiment also operates toreduce the number of frame buffer write transactions whenever possible.This is achieved by using the tile signature comparison process to alsocontrol whether a new tile is written to the frame buffer or not.

In particular, if the signature comparison process determines that thesignatures of the two tiles being compared match (i.e. the new tileshould be considered to be the same as the corresponding tile (the tilein the same tile position) that is already stored in the frame buffer),then the new tile is not written to the frame buffer, but instead theexisting tile is retained in the frame buffer for that tile position.

In this way, the present embodiment can avoid write traffic for sections(tiles) of the frame buffer that don't actually change from one frame tothe next. This can save a significant amount of bandwidth and powerconsumption in relation to the frame buffer operation.

On the other hand, if the signatures do not match, then the new tile iswritten to the frame buffer and the generated signature for the tile isalso written to memory.

This operation is achieved by the signature comparator 23, if it findsthat the tiles’ signatures do not match, controlling a write controller24 to write the new tile and its signature to the frame buffer andassociated signature data store in the memory 2. On the other hand, ifthe signature comparator 23 finds that the signature of the new tilematches the signature of the tile already stored in the frame buffer(i.e. that the tiles should be considered to be the same), then thewrite controller 24 invalidates the tile and no data is written to theframe buffer (i.e. the existing tile is allowed to remain in the framebuffer and its signature is retained).

In this way, a tile is only written to the frame buffer in the memory 2if it is found that by the signature comparison to differ from a tilethat is already stored in the memory 2. This helps to reduce the numberof write transactions to the memory 2 as a frame is being generated.

In the present embodiment, to stop incorrectly matched tiles fromexisting for too long a long period of time in the frame buffer, theability of the signature comparison to prevent a tile being written tothe frame buffer is periodically disabled (preferably once a second foreach stored tile in the frame buffer). This then means that when a tilefor which the ability of the signature comparison to prevent a tilebeing written to the frame buffer has been disabled is newly generated,the newly generated tile will inevitably be written to the frame bufferin the memory 2. In this way, it can be ensured that incorrectly matchedtiles will over time always be replaced with completely new (andtherefore correct) tiles in the frame buffer. In the present embodiment,the ability of the signature comparison to prevent a tile being writtento the frame buffer for the stored tiles is disabled in a predetermined,cyclic, sequence, so that each second (and/or over a set of say, 25 or30 frames), each individual tile position will always have a new tilewritten for it once.

In a particularly preferred embodiment, the tile signatures that aregenerated for use in the present invention are “salted” (i.e. haveanother number (a salt value) added to the generated signature value)when they are created. The salt value may conveniently be, e.g., theoutput frame number since boot, or a random value. This will, as isknown in the art, help to make any error caused by any inaccuracies inthe comparison process of the present invention non-deterministic.

Although in the above embodiment it is the display of the entire outputframe that is updated (or not), the present invention may also be usedto control the updating of only parts of the output display where thedisplay process supports such operation. This will be the case, forexample, for display interconnects that support partial frame updates.Equally, in some displays, the LCD controller can skip lines that havenot changed since the last frame by toggling a horizontal blank signalquickly without sending the data for the scan line. It would also bepossible to modify the interface between the display and the displaycontroller so as to be able to update only parts of the output framethat have been changed, if desired.

This could be used, for example, for displays with integrated memory, asthese displays will allow modifying parts of the display image at a timethrough the bus.

In this case, the tile comparison process could be used to assesswhether particular regions of the output frame have changed or not, andthen to control the updating of those particular regions only in themanner of the present embodiment.

For example, the comparison and control hardware could keep track ofwhich regions of the display of the output frame have been modified asbetween the output frames being compared, and force a display refreshfor those regions that have been modified, but otherwise refresh thedisplayed image at a low rate for regions of the image that remainunmodified. This would be particularly useful for displays that supportpartial updates.

In this case, the tile comparison process need not be carried out foreach and every tile that is generated, but may be done for the desiredregions of the display only, if desired.

Where the present invention is being used to update only portions of thedisplay in this fashion, then preferably the brightness of the portionof the image that is to be updated is modified (for example by changingthe gamma of the image portion in the display controller, or bycontrolling the backlight, and/or by manipulating the colours) so thatit has the same brightness as the rest of the image. This will help toprevent distortions to the image caused by updating only some but notall of the displayed image.

In the above embodiment, the rate at which the display 7 is updated fromthe frame buffer is controlled on the basis of the output framecomparisons. In another preferred embodiment of the present invention,the comparison and control hardware 5 is operable to vary the format inwhich the output frame is stored in the frame buffer on the basis of theoutput frame comparisons.

The Applicants have recognised that if the output frame is changingrapidly or significantly from frame to frame, the user viewing theframes will be unable to perceive much detail, whereas if the image isstatic, a higher quality image may be desirable. This second embodimentof the present invention exploits this by varying the frame bufferformat that is used in dependence upon whether the output framecomparisons indicate that the output frames are changing significantlyor are static.

FIGS. 4 and 5 show the configurations of the comparison and controlhardware 5 and the display controller 6, respectively, in thisembodiment. (The system of this embodiment is otherwise configured asshown in FIG. 1.)

As shown in FIG. 4, in this embodiment, the comparison and controlhardware 5 receives as its input tiles 25 from the graphics processor 1.Each tile is again then temporarily stored in a buffer 21 and passed toa signature generator 20. The buffer 21 buffers a complete frame, whichframe is then written out as a complete frame to the frame buffer in thememory 2 when the complete frame is stored in the buffer 21.

The signature generated for a new tile in the signature generator 20 isagain passed to a comparator 23, where it is compared to the signatureof the corresponding tile in the frame buffer (which may be retrievedfrom a signature buffer 22 or, if necessary, from the memory 2, asdiscussed above).

As in the preceding embodiment, if the comparator 23 finds that thesignatures do not match, it increments a counter 26 which keeps a countof the number of tiles that have been found not to match for the outputframe in question. This counter 26 is again reset to zero each time anew output frame is started, and then counts the number of unmatchingtiles for the frame being processed.

The count of mis-matching tiles for the current frame is again comparedto a predetermined threshold count value or tidemark in a comparator 27.If the count of mis-matching tiles for the output frame being comparedis found to exceed the threshold value, then it is assumed that the twooutput frames being compared are significantly different (i.e. the imageis changing significantly as between the two frames). This then triggersthe storing of the output frame in the frame buffer in a lower qualityformat. This process will now be described.

As shown in FIG. 4, the frame that is stored in the buffer 21 may eitherbe passed directly from the buffer 21 to the write controller 24 thatwrites the frame to the frame buffer in the memory 2, or it may bepassed to the write controller 24 via an encoder 40 that is operable toencode the frame from the buffer 21 into a different format.

In the present embodiment, it is assumed that the tiles are produced bythe graphics processor in an RGB 8:8:8 or RGB 10:10:10 format. This is ahigher quality format, but the additional quality can only really beperceived if the image is static. The encoder 40 therefore operates toconvert the frame (tiles) from this format into a lower quality, lossycompressed, format, in this case YUV 4:2:0 (which is often used forvideo where the image is constantly moving).

The write controller 24 is then controlled to store either the original,high quality format frame (tiles) directly from the buffer 21, or thelower quality format frame (tiles) from the encoder 40 by the thresholdcomparator 27. In particular, if the number of mis-matching tilesexceeds the threshold value, the write controller is triggered to storethe output from the encoder 40 in the frame buffer (i.e. such that thelower quality format is used) and vice-versa.

The threshold mis-matching tile value to trigger the use of the lowerquality format can be selected as desired. It should be a higher valuethan the value required to trigger a display update in the previousembodiment.

As shown in FIG. 4, as well as the output frame itself being written tothe frame buffer (in the selected format), a frame descriptor unit 41generates a description or identifier (meta-data) 42 that indicates theformat that the frame is being stored in. The frame format descriptor 42is then stored in association with the frame 43 in the memory 2 to allowthe display controller to identify the format that the frame has beenstored in in the frame buffer (as will be discussed further below). Theframe format descriptor information 42 may be an identifier indicatingthe format used, and/or an algorithm indicating how the data should beconverted to the desired display format, etc. Alternatively, the frameformat could be signalled to the display controller 6, if desired.

In an alternative arrangement, the comparator 27 could control thegraphics processor 1 to produce its tiles in the lower quality formatdirectly, where that is required.

FIG. 5 shows the corresponding display controller operation. As shown inFIG. 5, when the display controller receives a frame 51 from the framebuffer for display, it first checks the format descriptor 42 associatedwith the frame in question in a frame descriptor decoder 52. As a resultof that decoding, it then either passes the frame in the form it isreceived via a pass-through path 53 to the display controller logic fordisplay, or, if the input frame is in its encoded format, passes it to aframe decoder 54 for converting to the appropriate format for displaybefore again passing the frame to the display controller logic and thedisplay.

The display controller may conveniently, for example, read the frameformat descriptor during the VSYNC period immediately after the framebuffer swop before then reading and decoding the actual frame data. Thismakes it possible to straightforwardly switch frame buffer formatson-the-fly from frame to frame.

It can be seen therefore that in this embodiment the comparison andcontrol hardware 5 is operable to store the frame in the frame buffer ina desired format together with control information to indicate whichformat has been written. The display controller 6 then reads theinformation describing the frame buffer format, reads out the framebuffer data, and, if necessary, converts the frame buffer format to thedesired output format. The different formats to use are based on thecomparison of the output frames to see whether successive output framesdiffer significantly from each other or not.

This embodiment accordingly allows a lower image quality frame bufferformat to be used when the image is determined to be changing rapidly(such that the user will be unable to receive a great deal of detail andtherefore a lower image quality will be acceptable), thereby reducingpower consumption and bandwidth, but while still being able to revert toa higher quality image format when the image becomes static (and thusthe higher quality of the image can be perceived by the user).

By being able to dynamically alter the frame buffer format used based onthe rate of change of the image in this way, an arrangement whichimproves image quality compared with frame buffers that always use lowerquality format, but which also reduces memory bandwidth compared tosystems which would always use a higher quality frame buffer format, isprovided.

Although in the above embodiment, the higher quality format is describedas being RGB, and the lower quality format as being YUV, otherarrangements to provide higher and lower quality formats could be usedif desired. For example, other compressed, lossy formats could be usedfor the lower quality formats. It would also, for example, be possibleto use different frame buffer resolutions, and/or dynamic ranges (colourdepths), etc., to provide the higher and lower quality formats, ifdesired. Where the image is rendered at a lower resolution, then thedisplay controller or another system component could scale the image tofull size for display.

It would also be possible to provide only partial frame updates, e.g.such that only a region of each new frame is generated, rather thangenerating the entire new frame, where a lower quality format isdesired, with the higher quality format then being the generation of theentire new frame each time.

Equally, although the above embodiments are described with reference toselecting a particular format for the entire frame that is stored in theframe buffer, it would be possible to vary the format for particularregions of the frame only (i.e. to have different regions of the framestored in different formats). This could be done by using the tilecomparisons to identify those regions of the frame that are changingmore significantly and then using a different format for those regionsof the frame only, in a similar manner to the process discussed abovefor updating part of the display only.

In this case, the buffer 21 could, for example, simply store part of theframe, or part of the frame that is stored in the frame buffer 21 couldbe passed through the encoder 40 for writing to the frame buffer, withthe remainder of the frame not being so-encoded before it is written tothe frame buffer. Similarly, the frame descriptor 42 could be configuredto appropriately indicate the region or regions of the frame and whatformat they are encoded in.

Although the above embodiment has been described as being an alternativearrangement to the control of the display update rate based on the framecomparisons, as will be appreciated by those skilled in the art, itwould be possible to perform both the format selection and displayupdate rate control based on the frame comparisons if desired. Thus in apreferred embodiment, the frame comparisons are used to both control therate at which the display is updated and to select the format that theframe is stored in in the frame buffer. In this case two differentmis-matching tile count thresholds are preferably used, a first, lowerthreshold to trigger frame updates, and a second, higher threshold totrigger the use of a lower quality frame buffer format.

Although the present invention has been described above with particularreference to the controlling of the display update rate and/or the framebuffer format based on the comparisons of the output frames, thecomparisons of the output frames could be used to control other aspectsof the system operation as well or instead, if desired. For example, ifthe output frame comparisons show that the image is static for a periodof time, the graphics processor could be controlled to reduce the rateat which it produces the output frames, thereby saving power. Similarly,if it is found by the output frame comparisons that the output image isstatic for a period of time, then the system could be controlled tore-render the frame using better anti-aliasing, thereby increasing theperceived image quality.

A number of other alternatives and arrangements of the above embodimentsand of the present invention could be used if desired.

For example, it would be possible to provide hardware registers thatenable/disable the tile signature comparisons for particular frameregions, such that the signature generation and comparison is onlyperformed for a tile if the register for the frame region in which thetile resides is set.

The driver for the graphics processor (for example) could then beconfigured to allow software applications to access and set these tilesignature enable/disable registers, thereby giving the softwareapplication the opportunity to control directly whether or not and where(for which frame regions) the signature generation and comparisons takeplace. This would allow a software application to, for example, controlhow and whether the signature comparison is performed. This could thenbe used, e.g., to eliminate the power consumed by the signaturecomparison for a region of the output frame the application “knows” willbe static.

The number of such registers may chosen, for example, as a trade-offbetween the extra logic required implementing and using them and thedesired granularity of control.

Although the present embodiments have been described above withparticular reference to the comparison of rendered tiles to be writtento the frame buffer, as discussed herein it is not necessary that thedata blocks forming regions of the output data array that are compared(and e.g. have signatures generated for them) in the manner of thepresent invention correspond exactly to rendered tiles generated by thegraphics processor.

For example, the data blocks that are considered and compared in themanner of the present invention could be made up of plural renderedtiles and/or could comprise sub-portions of a rendered tile. Indeed,different data block sizes may be used for different regions of the sameoutput array (e.g. output frame) and/or the data block size and shapecould be adaptively changed, if desired.

Where a data block size that does not correspond exactly to the size ofa rendered tile is being used, then the comparison and control hardwareunit 5 may conveniently be configured to, in effect, assemble orgenerate the appropriate data blocks (and, e.g., signatures for thosedata blocks) from the data, such as the rendered tiles, that it receivesfrom the graphics processor (or other processor providing it data for anoutput array).

Although the present embodiments have been described above withparticular reference to a graphics processing system and the generationof output frames by a graphics processor, the Applicants have recognisedthat the frame comparison and control processes of the present inventioncould equally be used for other systems that generate output frames fordisplay, particularly where the output frame data is processed in blocksin a manner similar to the tiles of a tile-based graphics processor,such as a video processor (video codec) producing video blocks for avideo frame buffer, and for graphics processor image composition. Thusthe processes of the present invention may be applied equally to theimage that is being, for example, generated by a video processor.

It can be seen from the above that the present invention, in itspreferred embodiments at least, can help to reduce, for example, displayoperation power consumption and memory bandwidth.

This is achieved, in the preferred embodiments of the present inventionat least, by identifying opportunities to use lower display update ratesand/or less expensive frame buffer formats, etc., by comparing outputframes to determine whether the output frames are changing or not. Then,for example, if the output frame is only changing slowly or is static,the display is updated at a lower rate, thereby reducing powerconsumption and bandwidth, and if the output frame is changing rapidly,the image quality is reduced, thereby again reducing power consumptionand bandwidth.

The present invention, in its preferred embodiments at least, cansignificantly reduce the frame buffer and display controller powerconsumption and frame buffer bandwidth requirements. For example, whenthe display update rate is being controlled, the frame buffer accesspower saving is expected to be around 50% when the image is static.

For example, a 32-bit mobile DDR-SDRAM transfer consumes about 2.4 nJper 32-bit transfer. Thus, assuming a display update rate of 60 Hz,display refresh transactions will consume about (1920×1080×4)×2.4nJ/4×60=300 mW and 474 MB/s for HD graphics at 60 fps and(1024×768×4)×(2.4 nJ/4)×60=112 mW and 180 MB/s for 1024×768 graphics at60 fps.

Considering just the first order effect of frame buffer accesses andignoring on-chip interconnect, display control and video output powerconsumption, etc., if the display refresh rate is reduced from 60 fps to20 fps, that will save of the order of 200 mW and 316 MB/s for HDgraphics and 75 mW and 120 MB/s for 1024×768 graphics.

Thus the present invention is particularly applicable for use with lowerpowered devices and devices where power consumption is important. It istherefore particularly applicable for use in or with mobile and portabledevices.

1. A method of operating a data processing system in which a stream ofoutput frames to be displayed is generated by the data processing systemand written to a frame buffer for display on a display device, themethod comprising: comparing output frames to be displayed, andcontrolling at least one aspect of the way in which the output framesare provided for display on the display device on the basis of thecomparison.
 2. The method of claim 1, wherein the step of controlling atleast one aspect of the way in which the output frames are provided fordisplay on the display device on the basis of the comparison of theoutput frames comprises controlling the rate at which the display deviceis updated from the frame buffer on the basis of the comparison.
 3. Themethod of claim 2, further comprising: always updating the displaydevice at a minimum rate irrespective of the result of the framecomparisons.
 4. The method of claim 2, comprising updating only part ofthe frame displayed on the display device from the frame buffer on thebasis of the comparison of the output frames.
 5. The method of claim 1,wherein the step of controlling at least one aspect of the way in whichthe output frames are provided for display on the display device on thebasis of the comparison of the output frames comprises controlling theway that the output frame is stored in the frame buffer on the basis ofthe output frame comparisons.
 6. The method of claim 5, wherein the stepof controlling the way that the output frame is stored in the framebuffer on the basis of the output frame comparisons comprises selectingthe format that the frame is stored in the frame buffer in on the basisof the comparisons.
 7. The method of claim 6, further comprisingassociating with the output frame stored in the frame buffer controlinformation indicative of the format that the frame is stored in in theframe buffer.
 8. The method of claim 6, comprising using different framebuffer formats for different portions of a frame.
 9. The method of claim1, wherein the step of comparing the output frames to be displayedcomprises comparing blocks of data representing particular regions ofthe respective output frames with each other.
 10. The method of claim 9,wherein the data processing system is a tile-based graphics processingsystem and each data block that is compared corresponds to a renderedtile that the graphics processing system produces.
 11. A data processingsystem comprising: a data processor for generating a stream of outputframes to be displayed; a frame buffer for storing an output frame to bedisplayed; a write controller for writing an output frame generated bythe data processor to the frame buffer; a display device for displayingan output frame; a display controller for reading an output frame fromthe frame buffer and for providing it to the display device for display;and processing circuitry for comparing output frames to be displayed andfor controlling at least one aspect of the way in which the outputframes are provided for display on the display device on the basis ofthe comparison.
 12. The system of claim 11, wherein the processingcircuitry controls at least one aspect of the way in which the outputframes are provided for display on the display device on the basis ofthe comparison of the output frames by controlling the rate at which thedisplay device is updated from the frame buffer on the basis of thecomparison.
 13. The system of claim 12, wherein the display controllercomprises processing circuitry arranged to always update the displaydevice at a minimum update rate.
 14. The system of claim 12, wherein theprocessing circuitry can update only part of the frame displayed on thedisplay device from the frame buffer on the basis of the comparison ofthe output frames.
 15. The system of claim 11, wherein the processingcircuitry controls at least one aspect of the way in which the outputframes are provided for display on the display device on the basis ofthe comparison of the output frames by controlling the way that theoutput frame is stored in the frame buffer on the basis of the outputframe comparisons.
 16. The system of claim 15, wherein the processingcircuitry is arranged to select the format that the frame is stored inthe frame buffer in on the basis of the output frame comparisons. 17.The system of claim 16, further comprising processing circuitry forassociating with the output frame stored in the frame buffer controlinformation indicative of the format that the frame is stored in in theframe buffer.
 18. The system of claim 16, wherein the processingcircuitry can use different frame buffer formats for different portionsof a frame.
 19. The system of claim 11, wherein the processing circuitryis arranged to compare the output frames to be displayed by comparingblocks of data representing particular regions of the respective outputframes with each other.
 20. The system of claim 19, wherein the dataprocessing system is a tile-based graphics processing system and eachdata block that is compared corresponds to a rendered tile that thegraphics processing system produces.
 21. An apparatus for use in a dataprocessing system in which successive output frames to be displayed aregenerated by the data processing system and written to a frame bufferfor display on a display device, the apparatus comprising: processingcircuitry arranged to compare output frames to be displayed, and tocontrol at least one aspect of the way in which the output frames areprovided for display on the display device on the basis of thecomparison.
 22. The apparatus of claim 21, wherein the processingcircuitry controls at least one aspect of the way in which the outputframes are provided for display on the display device on the basis ofthe comparison of the output frames by controlling the rate at which thedisplay device is updated from the frame buffer on the basis of thecomparison.
 23. The apparatus of claim 22, wherein the processingcircuitry can update only part of the frame displayed on the displaydevice from the frame buffer on the basis of the comparison of theoutput frames.
 24. The apparatus of claim 21, wherein the processingcircuitry controls at least one aspect of the way in which the outputframes are provided for display on the display device on the basis ofthe comparison of the output frames by controlling the way that theoutput frame is stored in the frame buffer on the basis of the outputframe comparisons.
 25. The apparatus of claim 24, wherein the processingcircuitry is arranged to select the format that the frame is stored inthe frame buffer in on the basis of the output frame comparisons. 26.The apparatus of claim 25, further comprising processing circuitry forassociating with the output frame stored in the frame buffer controlinformation indicative of the format that the frame is stored in in theframe buffer.
 27. The apparatus of claim 25, wherein the processingcircuitry can use different frame buffer formats for different portionsof a frame.
 28. The apparatus of claim 21, wherein the processingcircuitry is arranged to compare the output frames to be displayed bycomparing blocks of data representing particular regions of therespective output frames with each other.
 29. The apparatus of claim 28,wherein the apparatus is for use in a tile-based graphics processingsystem and each data block that the apparatus compares corresponds to arendered tile that the graphics processing system produces.
 30. Acomputer readable storage medium storing computer software code forperforming the method of any one of claims 1 when the program element isrun on a data processor.